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Image registration of biological data is challenging as complex deformation problems are common. Possible 
deformation effects can be caused in individual data preparation processes, involving morphological 
deformations, stain variations, stain artifacts, rotation, translation, and missing tissues. The combining 
deformation effects tend to make existing automatic registration methods perform poor. In our experiments 
on serial histopathological images, the six state of the art image registration techniques, including TrakEM2, 
SURF + affine transformation, UnwarpJ, bUnwarpJ, CLAHE + bUnwarpJ and BrainAligner, achieve no 
greater than 70% averaged accuracies, while the proposed method achieves 91.49% averaged accuracy. The 
proposed method has also been demonstrated to be significantly better in alignment of laser scanning 
microscope brain images and serial ssTEM images than the benchmark automatic approaches (p < 0.001). 
The contribution of this study is to introduce a fully automatic, robust and fast image registration method 
for 2D image registration. 

Image registration is the process of transforming different sets of data into one coordinate system, and elastic 
image registration is potentially an enabling technology for the effective and efficient use of many image - 
guided diagnostic and treatment procedures, which rely on multimodality image fusion or serial image 
comparison. The use of elastic image registration covers a wide variety of medical applications, from building 
anatomical atlases, to longitudinal studies of tumor growth or other disease processes, through surgical planning 
and guidance Consequently, it is a creative field of research; techniques are numerous and inspired from a wide 
range of theories or techniques, such as statistics, information theory, theory of continuum mechanics, theory of 
thermodynamics, optical flow, splines, wavelets and block matching, and there are a number of reviews of 
biomedical image registration techniques and applications^"^. 

The histopathological study of tissue is an important tool in the medical field for the prognosis of disease. 
Conventionally, histological slides are examined under an optical microscopy to reveal two dimensional images. 
However, this method is insufficient to analyze complex three dimensional histology of lesion or tumor. By 
registering consecutive slices of biological images, 3D histology reconstruction can be generated^ In 2010, Peng 
et al.^ introduced a Vaa3D system for real-time visualization of a three-dimensional digital atlas of neurite tracts 
in the fruitfly brain. Another useful application is multimodal molecular mapping, requiring the joint analysis of 
two-dimensional gene and protein expression maps (obtained from in situ hybridization and immunohisto- 
chemistry, respectively) and/or their local comparison against cellular morphology maps (from histology) 3D 
reconstruction and visualization of microscopic images makes it possible to provide medical doctors and biol- 
ogists a rapid and precise data set about the patients or animals regarding the micro -environment and topology in 
the tissue or organ potentially involved in certain diseases or biological functions. 

Serial slides can be manually aligned by setting up a number of pairs of corresponding control points to the 
same (x, y) location for consecutive images Z/ and z^+i, and the pairs of images and paired-sets of control points 
are then given to semi-automatic software^ ^ for image alignment. Fully automatic registration of biological images 
is possible as demonstrated by software TrakEM2^^"^^ and in various studies^'^°'^^"^^. However, 3D reconstruction 
of histopathological data is challenging. Cardona et al.^^ pointed out that "TrakEM2 acknowledges that any 
automatic procedure (such as image registration and image segmentation) will eventually fail partially or fully and 
will require manual correction by a human operator". In comparison to the CT scans or confocal images as used 
in the studies^' where the serial image data maintains the property of geometrical continuity in 3d space, there 
are complex deformation problems for serial histopathological slides, including physical destructions caused by 
cutting and fixation, staining artifacts and uneven stain variations due to potential discrepancy in thickness of 
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Figure 1 | Image registration of histopathological images is difficult. 

Image registration is the process of transforming different sets of data into 
one coordinate system, but for registering histopathological images, there 
are challenging morphological deformation problems (as displayed in the 
left pair) in combination with staining variation and staining artifacts (as 
illustrated in the right pair). 

individual tissue sections. These complex distortion effects makes 
image registration of histological data a difficult task (see Figure 1). 
In previous studies^'^°'^^"^^'^^, the raw data tends to be carefully pre- 
pared, showing little morphological distortions or stain variation, to 
simplify the registration task. In addition, they tend to work with low 
resolution images without dealing with high precision tissue or cell 
level registration. Figure 2 illustrates possible deformation problems 
caused in individual data preparation processes, involving morpho- 
logical deformations, stain variations, stain artifacts, rotation, trans- 
lation, and missing tissues. The combining deformation effects tend 
to make existing automatic registration methods perform poor. 

The main contribution of this study is to introduce a fully auto- 
matic and robust image registration method that is able to deal with 
complex deformation problems commonly found in biological 
imagery. For evaluation, three types of data are utilized, including 
47 pairs of serial histopathological images, 30 laser scanning micro- 



scope images^^ and 30 serial ssTEM images^^. The 47 pairs of serial 
histopathological images were collected under routine sample pre- 
paration procedures in hospitals, showing the above-mentioned 
deformations. The proposed method is compared with six existing 
image registration techniques, including an elastic b-spline model for 
biological images (UnwarpJ)^°, an improved bi-directional b-spline 
model for histopathological section alignment (bUnwarpJ)^^ a popu- 
larly adopted image registration approach (SURF + affme trans- 
formation) a serial section registration approach^^ using the 
software package (TrakEM2)^^"^^, BrainAligner^^ and a combination 
of a local contrast correction method (CLAHE)^^ and bUnwarpJ. The 
results show that the existing image registration techniques perform 
poorly on the serial histopathological data with deformations and 
achieve less than the accuracy 70% on average while the proposed 
method achieves the averaged accuracy 91.49%. In alignment of laser 
scanning microscope brain images the results show that the pro- 
posed method is significantly better than the automatic Brain- 
Aligner^^ (p < 0.001). In registration of serial ssTEM images^'*, the 
proposed method is also significantly better than TrakEM2^^ {p < 
0.001). 

This work combines and extends our previous efforts published in 
Wang and Chen^^, which integrates the strengths of area-based and 
feature-based approaches and uses sparse approximation for coarse 
and fast global registration to greatly improve the performance of 
image registration. As we observed that data normalization and fea- 
ture enhancement are vital for the following global feature-based 
registration and local area-based registration, we further improve 
the registration method by modifying the data normalization and 
feature enhancement parts. The proposed method is able to not only 
reduce global and local variations among slides but also effectively 
identify corresponding features, and with accurately identified land- 
mark features and normalized data, the method produces valid regis- 
tration outputs robust to various deformation problems commonly 
occurred in biological data, such as morphological distortions, stain- 
ing variations, staining artifacts and loss of tissue. 

Results 

Serial histopathological image registration. 47 pairs of consecutive 
slides were collected under routine sample preparation procedures in 
hospitals using C57 mice with IgAN (Immunoglobulin A (IgA) 
nephropathy), which is the most common glomerular disorder 
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Figure 2 | Challenges of registration of histopathological images. The challenges of registration of histopathological images include complex geometrical 
deformation and stain variations induced from individual data preparation steps. 
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Table 1 Quantitative 


evaluation 


on Histopathological Images 
















Score 




95% C.L for Mean 


Method 


N 


Mean Std. Deviation 


Std. Error 


Lower Bound 


Upper Bound 


120 

UnwarpJ 


4/ 


58.72 


44.99 


O.OO 


45.51 


71.93 


bUnwarpJ^^ 


47 


59.15 


44.32 


6.47 


46.14 


72.16 


SURF22 


47 


14.89 


22.25 


3.25 


8.36 


21.43 


TrakEM2^2-i5 


47 


51.06 


43.9 


6.4 


38.17 


63.95 


CLAHE24 + bUnwarpJ2i 


47 


69.36 


43.51 


6.35 


56.59 


82.14 


Wang25 


47 


89.36 


28.55 


4.16 


80.98 


97.74 


Proposed Method 


47 


91.49 


23.12 


3.37 


84.7 


98.28 




Table 2 LSD Multiple Comparison-Histopathological Images 


LSD 










95% C. 




(l)Method 


(J)Method 


Mean Difference (l-J) 


Std. Error 


Sig. 


Lower Bound 


Upper Bound 


Proposed 


UnwarpJ 


32.77* 


7.92 


<0.001 


17.18 


48.35 


Method 


bUnwarpJ 


32.34* 


7.92 


<0.001 


16.75 


47.93 


(cw-R) 


SURF 


76.6* 


7.92 


<0.001 


61.01 


92.18 


TrakEM2 


40.43* 


7.92 


<0.001 


24.84 


56.01 


CLAHE + bUnwarpJ 22.13* 


7.92 


0.006 


6.54 


37.71 




Wang 


2.13 


7.92 


0.774 


-12.48 


16.73 


*The proposed method is significantly better than 


the benchmark techniques using the LSD test (p < 0.01 ). 









across the world^^. IgAN was induced by daily injection of purified 
IgA anti-phosphorylcholine and pneumococcal C-polysaccharide 
(PnC)^^, and all animal experiments were performed with the 
approval (permit number IACUC-1 1-063) by the institutional 



animal care and use Committee of the national defense medical 
center, Taiwan. The tissues were fixed in 10% buffered formalin 
and embedded in paraffin, and serial sections (4 ^m) were cut 
using Leica RM2155 and stained with hematoxylin and eosin 
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Figure 3 | A box plot of quantitative evaluation results of histopathological image registration. The presented method works constantly well overall and 
outperforms the benchmark approaches (see Table 1). Outliers > 1.5 X interquartile range are marked with a dot, and outliers > 3 X interquartile range 
are marked with an asterisk. 
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Figure 4 | Quantitative evaluation of image registration for the left image pair in Figure 1. The blue rectangles represent the locations of the selected 
landmarks defined by experienced pathologists in the target image(a); the red boxes represent mismatches of corresponding landmarks in the 
transformed source image; the green boxes represent matches of corresponding landmarks in the transformed source image. There is no match for the 
transformed source images by (c)UnwarpJ, (d) bUnwarpJ, and (e)SURF, and only one match for (f)TrakEM2. In comparison, the proposed method(g) 
aligns the images well, achieving 100% target and transformed source matching successful rate. 



(H&E). Images were then captured from glass slides with standard 
brightfield light microscopy (Olympus, Japan) at the magnification 
of 400 X. 

Regarding the evaluation method, as pointed out in a recent 
study^^ that conventional evaluation methods based on sum of 
squared differences (SSD) of image intensities between the target 
and the transformed source (see Equation 1) to represent the regis- 
tration accuracy level can be misleading for tissue image applications 
not only because that the intensity value of the pixel in the target and 
the one of the wrongly registered pixel in the transformed source may 
appear similar due to similarity of local tissue image features but also 
because that the intensity of the pixel in the target and the one of the 
accurately registered pixel in the transformed source may appear 
different due to stain variation. Hence, the quantitative evaluation 
approach of the previous work^^ is adopted; five corresponding land- 
marks between target images and associated transformed source 
images by each registration method were firstly manually marked 
by experienced pathologists, and an automatic matching system is 
built to compare the coordinates of the corresponding landmarks. 
The registration accuracy for each image pair is computed by the 
matching successful rate over the corresponding landmarks (within a 
five pixel distance), and the performance of each registration method 
is evaluated by the averaged accuracies over all image pairs. 



Table 1 presents the quantitative evaluation results on registration 
accuracies. The benchmark approaches^^"^^'^""^^ all perform poor, 
achieving less than 70% accuracy on average, and hence manual 
intervention are frequently required for accurate alignment. In com- 
parison, the proposed method obtains registration accuracy 
(91.49%), greatly outperforming the benchmark methods and show- 
ing slight improvements than the author's previous work^^. Using 
SPSS software^^, the statistical analysis results in Table 2 show that 
the proposed method is significantly better than the benchmark 
techniques {p < 0.01) based on the Fisher's Least Square 
Difference test (LSD). Figure 3 presents the box plot of the quant- 
itative scores, showing that the presented method frequently achieves 
higher scores than the benchmark methods. Figure 4 illustrates the 
quantitative evaluation results of alignment of the left image pair in 
Figure 1, showing that the proposed method (Fig. 4(h)) aligns the 
images well, achieving 100% matching score while the benchmark 
methods perform poor due to complex deformation problems, stain- 
ing variations and confusing local image features of biological data. 
Figure 5 shows another image registration example by the proposed 
method and the benchmark techniques. For the benchmark meth- 
ods, the parameters adopted in the experiments are the same para- 
meters used in the reference works^^'^°'^\ and the parameter values 
are listed in the method section. 



SCIENTIFIC REPORTS | 4 : 6050 | DOI: 1 0.1 038/srep06050 



4 



Source Imase 




Transfonn the source image into the 
coordinate system of the target image 

Image Registration 



Target Image 



Methods 

I 

Transfoiination Function 
(Deforaiation Field) 

Transfonned Source Image 




(a)UnwaipJ 



(b)BunwaipJ (c)CLAHE+Bunwai-pJ (d)SURF 



(e)TrakEM2 (f)Proposed Method 




Figure 5 | Performance comparison on registration methods. Existing image registration methods perform poor on the histopathological image pair 
with complex distortion problems while the proposed method is able to produce a valid transformation output, (a) the transformed source image of an 
elastic b-spline method^" designed for biological images, (b) the output of a bi-directional b-spline approach^^ (c) the transformed source image by 
CLAHE^^ + bUnwarpJ, (d)the registration output of a popularly adopted registration method, i.e. SURF^^, (e) the output of TrakEM2^^"^^ (f) the 
transformed source image of the proposed technique. 



Laser scanning microscope images of neurons from different 
brains. BrainAligner^^ is an automatic computer program that 
registers pairs of 3D image stacks using landmark matching. In 
registration of the histopathological images, BrainAligner fails to 
produce valid alignment outputs. We further compare the 
performance of the proposed method and BrainAligner based on 
the data used in the BrainAligner study^^. The test data are 30 laser 
scanning microscope images of the Drosophila and downloaded 
from http://penglab.janeUa.org/proj/brainaligner/BrainAligner/ 
Download.html. Regarding the parameters used in BrainAligner, 
all parameters are based on the system default settings. For fully 
automatic registration (without inputs of the manually marked 
corresponding landmarks), both global and local registration types 
are tested in the experiments. 

Unlike histopathological images, local image features of laser scan- 
ning brain images appear less confusing to one another, and there- 
fore a general registration performance measurement method, i.e. 
the percentage of pixels with similar intensity levels, is adopted to 
measure the registration accuracy, and an automatic evaluation tool 
is built to conduct quantitative evaluation automatically. The regis- 
tration accuracy, r, is formulated as follows. 

#{x:\\h{x)-U{h{x))<t} 

#Q ^ ' 

where 7i, I2, U{Ii) represents the source, target and transformed 
source images; Q= {x e Qif^Z^ : d(x) e ^2 0^^} defines a mask 



common to the source and target images, and #Q is the size of the 
mask in pixels; t = 30 in our experiments. 

In our experiments, it is observed that with manually marked 
corresponding landmarks between the source and target images, 
semi-automatic BrainAligner performs well. However, without 
manually predefined landmarks, fully-automatic BrainAligner does 
not produce good registration outputs. In comparison, the proposed 
fully automatic method without manual inputs performs well. The 
registration accuracy scores are summarized in Table 3, and the 
statistical analysis of LSD test is displayed in Table 4, showing that 
the presented fully- automatic method is significantly better than the 
fully- automatic BrainAligner {p < 0.001). The notable advantage of 
the presented method is to save costs of manual efforts and time in 
labelling landmarks. Figure 6 presents the box plot of the registration 
accuracy scores, showing that the presented method frequently 
achieves high scores in comparison to the benchmark approaches. 
Figure 7 displays the registration outputs of six image pairs by the 
proposed fully- automatic method, the fully- automatic BrainAligner 
and the semi-automatic BrainAligner (with manually predefined 
landmarks). 

A comparison on computational speed among the proposed 
method, the semi-automatic BrainAligner and the fully- automatic 
BrainAligner is shown in Figure 8. The processing time for the semi- 
automatic BrainAligner does not include the time for the manual 
marking of corresponding landmarks as this information is 
unknown in this study. Since the manual marking time is often 



Table 3 | Quantitative evaluation on brain images: Proposed method versus BrainAligner 



Score 



Method 



N 



Mean 



Std. Deviation 



Std. Error 



BrainAligner^^ without predefined landmarks 30 53.04 8.84 1.61 

BrainAligner^^ with predefined landmarks 30 78.41 3.41 0.62 

Proposed Method (cw-R) 30 78.15 3.55 0.64 
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Table 4 | 


LSD Test: Proposed method versus BrainAligner 








LSD 


(l)Method 


(J)Method 


Mean Difference (l-J) 


Std. Error 


Sig. 


Proposed 
Method 


BrainAligner without 
predefined landmarks 

BrainAligner with predefined 
landmarks 


25.11* 
-.25 


1.50 
1.50 


<0.001 
0.86 


*The proposed method is significantly better than the benchmark techniques using the LSD test (p < 0.01 ). 



lengthy, therefore the processing time by the semi-auto BrainAHgner 
might be greatly underestimated. Comparing the automatic 
approaches, on average the proposed method takes 3.79 second per 
registration while the automatic BrainAligner with the local registra- 
tion mode costs 22.93 second per registration on average, and the 
automatic BrainAligner with the global registration mode costs 52.32 
second per registration on average. In statistical analysis, the pre- 
sented method is significantly faster than the automatic BrainAligner 
approach {p < 0.001 based on Tukey HSD and LSD tests). 

Serial section transmission electron microscopy (ssTEM) of dro- 
sophila neural tissue. We have also compared the performance of 
the proposed method and TrakEM2 based on the data used in the 
TrakEM2 study^'*. The test images are 30 serial ssTEM sections of the 
Drosophila first instar larva ventral nerve cord (VNC) and 
downloaded from http://www.ini.uzh.ch/~acardona/data.html. In 



evaluation, the general registration performance measurement 
method, i.e. the percentage of pixels with similar intensity levels, is 
adopted to measure the registration accuracy (see Equation 1). 

Figure 9 displays registration results of six pairs of ssTEM images 
by the proposed method and TrakEM2; row (a) and (b) show target 
images and source images, respectively, and row (c) and (d) present 
registered source images by the proposed method and TrakEM2, 
respectively. It is observed that in this data set, there is no distinctive 
morphological deformation between the target images(a) and the 
source images (b), but illumination variations between the source 
and the target can be large, which tends to make the overall registra- 
tion scores lower in comparison to the BrainAligner data set. In the 
second column of Figure 9, it shows that TrakEM2 generates poorer 
alignment output(d) than the proposed method(c). Figure 10 dis- 
plays the box plot of the registration accuracy scores by the proposed 
method and TrakEM2, showing that the proposed method tends to 
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Figure 6 | A box plot of registration accuracy scores on laser scanning brain images from BrainAligner study^^. 100 indicates that the method accurately 
align the target and source images, and 0 indicates poor registration. 
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Figure 7 | Registration of brain images by brainAligner and proposed method. The brainAligner performs well with predefined landmarks but poor 
without predefined landmarks. In comparison, the proposed method without predefined landmarks produces accurate alignment outputs, (a) the target 
brain images, (b) the source brain images, (c) the registered source images by the proposed method (cw-R), (d) the registered source images by 
brainAligner without predefined landmarks (under the local registration mode), (e) the registered source images by brainAUgner without landmarks 
(under the global registration mode), (f) the registered source images by brainAligner with predefined landmarks. 



obtain higher scores than TrakEM2. Moreover, using ANOVA ana- 
lysis, the proposed method is significantly better than TrakEM2 in 
image registration of serial ssTEM images (p < 0.001). 

Discussion 

We present a fully automatic elastic registration method for aligning 
2D tissue images robust to morphological distortions, staining var- 
iations and staining artifacts. In addition, the presented method for 
detection of corresponding landmarks works reliably without man- 
ual intervention, and the presented image registration algorithm is 
not only limited to histopathological slides but can also be applied to 
other anatomically or histologically defined biological images such as 
laser scanning microscope brain images and serial ssTEM images. 
Moreover, as complex deformations are unavoidable in real life data, 
the presented technique will prove to be a substantial advantage for 
any application that requires image registration. We have integrated 
our method into a public Java-based image processing system, 
ImageJ^^, and the software is made publicly available (http://www- 
o.ntust.edu.twZ—cweiwang/ImageRegistration/), allowing the sci- 



entific community to download and use the robust image registration 
system (Cw-R). 

Methods 

This paper presents a fully automatic, robust and fast registration method that inte- 
grates the strengths of both area-based approaches and feature-based methods for 
biological data, containing three main parts: (1) improved data normalization and 
feature extraction, (2) sparse approximation of images for coarse and fast global 
registration, (3) Optimize and refine local registration by area-based direct matching 
approach. This work combines and extends our previous efforts published in Wang 
and Chen^^ and the improvement is made on the first part, which is improved data 
normalization and enhanced feature extraction. In this section, descriptions are 
focused on the modified part, and other parts can be found in the previous 
publication^^ 

Improvements on data normalization and image feature extraction can not only 
help identify more effective corresponding feature pairs for global registration but 
also boost up the performance of area-based local registration. Figure 1 1 shows the 
results of detection of corresponding feature pairs by SURF, SIFT, the author's pre- 
vious work^^ and the proposed method, showing that (a) SURF produces poor 
matching, (b)SIFT generates some incorrect matches, and (d)the proposed method 
find more accurately- matched corresponding feature pairs for fast and higher level 
coarse registration than (c)the author's previous work^^. 
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Figure 8 | A box plot of the image registration processing speed on the brain images. The processing time (in seconds) on brain image registration by the 
proposed method, the semi-automatic brainAligner and the automatic brainAligner. The processing time for the semi-automatic brainAligner does not 
include the time for manually marking corresponding landmarks as this information is unknown in this study. Since the manual marking time is often 
lengthy, therefore the processing time by the semi-automatic brainAligner is greatly underestimated. Comparing the automatic approaches, the proposed 
method is significantly faster than the brainAligner approach (p < 0.001 based on Tukey HSD and LSD tests) and costs 3.79 second per registration. 



Improved data normalization and feature extraction. An improved data 
normalization process is proposed to reduce variations on image features and 
enhance tissue patterns. This greatly benefits global feature matching and local area- 
based directing matching processes. It automatically optimizes the brightness and 
contrast of individual color channels based on the analysis of the image histogram 
distribution. Furthermore, as common contrast stretching algorithms are sensitive to 
spike, ridges or long tails problems at the tails of the histogram distribution and 
unable to detect the dynamic range of the main cluster, a modified method is 
presented here to effectively identify the dynamic range of the main cluster and deal 
with spike, ridges or long tails problems. Figure 12 presents (a) a raw tissue image with 
long tail problem, (b) the image generated by a commonly adopted auto contrast 
adjustment method (using Adobe Photoshop) and (c) the image by the presented data 
normalization method. Figure 12 shows that tissue image features such as nuclear 
patterns are greatly enhanced by the presented method. 

In conventional histopathological staining (H&E), Hematoxylin induces the 
blue staining of nuclei, and Eosin induces the red/pink staining of cytoplasm. 
Based on our previous study^° showing that applying histogram equalization in 
RGB color space performed better in separating the nuclei from the cytoplasm 
than in HSL color space, both to enlarge the difference between the nuclear and 
cytoplasmic expression in color space and to further produce more distinctive 
tissue image features, the normalization is applied to the Red, Green, and Blue 
components of each image. 

In color images, the value of each pixel is represented by a vector X with 
elements the pixel values of each color component. Assuming 

=X= [x'^\x'^^ ,x^^] a random vector, which models the pixel value for each 
color component Ci, C2, C3 in a color image. Firstly, the lower and the upper bound 
intensity levels of the histogram of each channel, Xiow and Xhigh, are computed by 
the equations below. Given a histogram distribution H, where H{x) is the number 
of pixels with intensity level x, the lower and the upper bound values for trans- 
formation are formulated as follows. 



argmin 5]H(^)'^^ * ^<H(x)^^ < 5]H(^)^^ 



(2) 



= arg max 



k 



(3) 



where cf. {ci, C2, C3}, a = 0.1 and = 0.002. 

Next, it maps the original pixel value x'^' in the range from x^^^^ to x^,^^ to new value 
x^i in the valid intensity scale from to Xmax- 



X ^ > Xmax 



(4) 



The data normalization greatly reduces stain variation and enhances tissue patterns, 
assisting the following feature extraction model to identify valid corresponding 
landmarks and improving image registration outcomes. An illustration is given in 
Figure 13; (a) the source image, (b) the target image, (c) and (d) are the images applied 
with the presented data normalization method with automatically identified corres- 
ponding landmarks, (e) and (f) are images with deformation grid, (g) is the 
deformation field, and (h) is the transformed source image that is well aligned to the 
target image. The tissue patterns of the images applied with the presented data 
normalization method are greatly enhanced, assisting the following feature extraction 
model to identify valid corresponding landmarks and improving image registration 
outcomes. 

For biological images, although the dyes used are visualized as having different 
colors, the resulting stains actually have complex overlapping absorption spectra. In 
the previous studies, color deconvolution was used to achieve color separation in 
forensic image processings^ and to achieve stain separation^^'^^ in biological image 
processing. Our goal is to extract the eosinophilic structures, which are generally 
composed of intracellular or extracellular protein, as image features for image 
registration, and the color decomposition technique is utilized to extract independent 
haematoxylin and eosin stain contributions from individual histopathological images 
using orthonormal transformation of RGB. 
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Figure 9 | Registration of ssTEM images by TrakEM2 and the proposed method, (a) the target images, (b) the source images, (c) the transformed source 
images by the proposed method (cw-R), (d) the transformed source images by TrakEM2. 
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Figure 10 | A box plot of registration accuracy scores on ssTEM images from TrakEM2 study^^. 100 indicates that the method accurately aUgn the target 
and source images, and 0 indicates poor registration. 
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(b)SIFT 





(c) Wang and Chen 



(d) proposed method 



Figure 11 | Detection of corresponding features by SURF, SIFT, the author's previous work^^ and the proposed method. (a)SURF produces poor 
matching, (b)SIFT generates some incorrect matches, and (d)the proposed method find more accurately-matched corresponding feature pairs for fast 
and higher level coarse registration than (c)the author's previous work. 



In the RGB color-space, every color is defined as c= (ci,C2,C3) = {r,g,b) where r,g, b 
represent the red, green and blue components, and we can see additive color mixing as 
the vector addition of RGB components. To model the colors in an image as the vector 
addition of a desired (D) and undesired (U) components to a background color (P), 
new unit vectors can be defined as follows. 



u = PU 



d=PD 



n = u X d 



(5) 



(6) 



(7) 



where n is perpendicular to u and d; n, u, d span the 3D space; PU and PD are 
alternative unit vectors based on the undesired and desired colors. 



Then, color c can be transformed to the new unit vectors. 

c = r-r+g-g + b-b = u-u + d-d + n-n+p 



(8) 



where p= OP; O is the origin in the RGB 3D space; OP is a vector. 

By setting w = 0, we remove the undesired component and obtain the new color 

c' =d-d + n-n+p.ln the case of three channels, the color system can be described as a 
matrix of the form with every row representing a specific stain and every column 
representing the optical density (OD) as detected by the red, green and blue channel 
for each stain. 



/ Cii Ci2 Ci3 \ 
M= C21 C22 C23 

\ C31 C32 C33 / 

For normalization, each OD vector is divided by its total length, such that 



(9) 




Figure 12 | Comparison of data normalization methods. The proposed data normalization method is able to deal with spike, ridges or long tails 
problems at the tails of the histogram distribution and effectively enhance tissue patterns, (a) a raw tissue image with long tail problem, (b) image by a 
commonly adopted auto contrast adjustment method (Adobe Photoshop is used here), (c) image by the presented data normalization method where 
tissue image features such as nuclear patterns are enhanced. 
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Figure 13 | Illustration of the proposed data normalization method. The proposed data normalization method enhances tissue patterns, reduces stain 
variation, and assists the following identification of corresponding landmarks and image registration, (a) the source image, (b) the target image, (c) and 
(d) are the images applied with the presented data normalization method with automatically identified corresponding landmarks, (e) and (f) are images 
with deformation grid, (g) is the deformation field, and (h) is the transformed source image that is well aligned to the target image. 



= C31 1 \J C31 +C32 + C33). In this study, the normalized optical density (OD) 

matrix, M, to describe the color system for orthonormal transformation is defined as 
follows: 

/RGB \ 

0.6442 0.7166 0.2668 Haematoxylin , , 

M= ^ 10 

0.0928 0.9541 0.2831 Eosin 

\ 0 0 0 / 

When C is the 3X1 vector for amounts of the stains at a particular pixel, the vector of 
OD levels detected at that pixel is equal to L = CM. Therefore, multiplication of the 
OD image with the inverse of OD matrix results in orthogonal representation of the 
stains forming the image (C = M-^L). Then, the image features of the red channel are 
extracted as eosinophilic structures for both high level feature-based coarse regis- 
tration and local area-based direct matching registration. 



Parameter values used in the experiments. For the benchmark methods, the 
parameters used in the experiments are the default parameters in the 
implementations of the reference works and they are the same parameters as 
used in the previous studies^'^'^"'^^ The values of the parameters for the benchmark 
methods and the proposed method are listed below. 

UnwarpJ parameter values. Initial deformation = Very Coarse, Final deformation = 
Fine, Divergence weight = 0.0, Curl Weight = 0.0, Landmark Weight = 0.0, Image 
Weight = 1.0, Stop Threshold = 0.01. 

bUnwarpJ parameter values. Registration Mode = Accurate, Initial deformation = 
Very Coarse, Final deformation = Fine, Divergence weight = 0.0, Curl Weight = 0.0, 
Landmark Weight = 0.0, Image Weight = 1.0, Stop Threshold = 0.01. 

TrakEM2 parameter values, mode = least squares (linear feature correspondences). 
For Scale Invariant Interest Point Detector, initial gaussian blur = 1.60px, Steps per 
scale octave = 3, minimum image size = 64px, maximum image size = 600px. For 
Feature Descriptor, feature descriptor size = 8, feature descriptor orientation bins = 
8, closest/next closest ratio = 0.92. Maximal alignment error = lOOpx, minimal inlier 
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ratio = 0.2, minimal number of inliers = 7, expected transformation = rigid, 
tolerance = 0.5px. Desired transformation = Rigid, correspondence weight = 1.00. 
In optimization, maximal iterations = 2000, maximal plateauwidth = 200, mean 
factor = 3.00. 

SIFT parameter values used in the proposed method, initial gaussian blur = 1 .05px, 
Steps per scale octave = 3, minimum image size = 64px, maximum image size = 
1024px, feature descriptor size = 4, feature descriptor orientation bins = 8, closest/ 
next closest ratio = 0.92, maximal alignment error = 30px, minimal inlier ratio = 

0. 05. minimal number of inliers = 7, expected transformation = rigid. 

Software and test images. The developed software is platform independent and thus 
can be executed in different operation systems such as Windows, Linux or Mac. The 
software with some test images can be downloaded from the author's website (http:// 
www-o.ntust.edu.tw/~cweiwang/ImageRegistration/). 
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